Advanced Lane Finding Project¶
The goals / steps of this project are the following:
- Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
- Apply a distortion correction to raw images.
- Use color transforms, gradients, etc., to create a thresholded binary image.
- Apply a perspective transform to rectify binary image ("birds-eye view").
- Detect lane pixels and fit to find the lane boundary.
- Determine the curvature of the lane and vehicle position with respect to center.
- Warp the detected lane boundaries back onto the original image.
- Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.
In [1]:
import numpy as np
import cv2
import matplotlib.pyplot as plt
from scipy.signal import find_peaks_cwt
import matplotlib.image as mpimg
import glob
%matplotlib inline
First, I'll compute the camera calibration using chessboard images¶
In [3]:
#Camera Calibration
images=glob.glob('./camera_cal/calibration*.jpg')
objpoints = [] #3D points in real world space
imgpoints = [] #2D points in image plane
objp = np.zeros((6*9,3), np.float32)
objp[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2)
for fname in images:
img = mpimg.imread(fname)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
ret, corners = cv2.findChessboardCorners(gray,(9,6), None)
if ret==True:
imgpoints.append(corners)
objpoints.append(objp)
img = cv2.drawChessboardCorners(img,(9,6), corners, ret)
#plt.imshow(img)
else:
print('No corners found!')
print(fname)
img_size = (img.shape[1], img.shape[0])
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, img_size, None,None)
Determine the region in the images that contains the lanes¶
In [4]:
def region_of_interest(img, vertices):
"""
Applies an image mask.
Only keeps the region of the image defined by the polygon
formed from `vertices`. The rest of the image is set to black.
"""
#defining a blank mask to start with
mask = np.zeros_like(img)
#defining a 3 channel or 1 channel color to fill the mask with depending on the input image
if len(img.shape) > 2:
channel_count = img.shape[2] # i.e. 3 or 4 depending on your image
ignore_mask_color = (255,) * channel_count
else:
ignore_mask_color = 255
#filling pixels inside the polygon defined by "vertices" with the fill color
cv2.fillPoly(mask, vertices, ignore_mask_color)
#returning the image only where mask pixels are nonzero
masked_image = cv2.bitwise_and(img, mask)
return masked_image
Edge detection - use color transforms, gradients, etc., to create a thresholded binary image.¶
To determine the edges in the picture I am using the following techniques to extract the lanes:
- Gaussian Blurring to reduce noises
- Sober Operator x-direction
- Sober Operator y-direction
- Direction of Gradient
- Magnitude of Gradient
- HLS Color Threshold
In [5]:
#calculate the camera matrix and distortion coefficients
def cal_undistort(img):
# Use cv2.calibrateCamera and cv2.undistort()
#undist = np.copy(img) # Delete this line
undist= cv2.undistort(img, mtx, dist, None, mtx)
return undist
def abs_sobel_thresh(gray, orient='x', sobel_kernel=3, thresh=(0, 255)):
# Convert to grayscale
#gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
# Apply x or y gradient with the OpenCV Sobel() function
# and take the absolute value
if orient == 'x':
abs_sobel = np.absolute(cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel))
if orient == 'y':
abs_sobel = np.absolute(cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel))
# Rescale back to 8 bit integer
scaled_sobel = np.uint8(255*abs_sobel/np.max(abs_sobel))
# Create a copy and apply the threshold
binary_output = np.zeros_like(scaled_sobel)
# Here I'm using inclusive (>=, <=) thresholds, but exclusive is ok too
binary_output[(scaled_sobel >= thresh[0]) & (scaled_sobel <= thresh[1])] = 1
# Return the result
return binary_output
def mag_thresh(gray, sobel_kernel=3, mag_thresh=(0, 255)):
# Convert to grayscale
#gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
# Take both Sobel x and y gradients
sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel)
sobely = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel)
# Calculate the gradient magnitude
gradmag = np.sqrt(sobelx**2 + sobely**2)
# Rescale to 8 bit
scale_factor = np.max(gradmag)/255
gradmag = (gradmag/scale_factor).astype(np.uint8)
# Create a binary image of ones where threshold is met, zeros otherwise
binary_output = np.zeros_like(gradmag)
binary_output[(gradmag >= mag_thresh[0]) & (gradmag <= mag_thresh[1])] = 1
# Return the binary image
return binary_output
def dir_threshold(gray, sobel_kernel=3, thresh=(0, np.pi/2)):
# Grayscale
#gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
# Calculate the x and y gradients
sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=sobel_kernel)
sobely = cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=sobel_kernel)
# Take the absolute value of the gradient direction,
# apply a threshold, and create a binary image result
absgraddir = np.arctan2(np.absolute(sobely), np.absolute(sobelx))
binary_output = np.zeros_like(absgraddir)
binary_output[(absgraddir >= thresh[0]) & (absgraddir <= thresh[1])] = 1
# Return the binary image
return binary_output
In [6]:
def color_transform(img):
# Choose a Sobel kernel size
img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
ksize = 7 # Choose a larger odd number to smooth gradient measurements
gradx = abs_sobel_thresh(img, orient='x', sobel_kernel=ksize, thresh=(50, 130))
grady = abs_sobel_thresh(img, orient='y', sobel_kernel=ksize, thresh=(50, 130))
mag_binary = mag_thresh(img, sobel_kernel=ksize, mag_thresh=(20, 80))
dir_binary = dir_threshold(img, sobel_kernel=ksize, thresh=(0, np.pi/2))
combined = np.zeros_like(dir_binary)
combined[((gradx == 1) & (grady == 1)) | ((mag_binary == 1) & (dir_binary == 1))] = 1
return combined
def color_HLS_transform(img):
# Convert to HLS color space and separate the S channel
# Note: img is the undistorted image
hls = cv2.cvtColor(img, cv2.COLOR_RGB2HLS)
s_channel = hls[:,:,2]
# Grayscale image
# NOTE: we already saw that standard grayscaling lost color information for the lane lines
# Explore gradients in other colors spaces / color channels to see what might work better
gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
# Sobel x
sobelx = cv2.Sobel(gray, cv2.CV_64F, 1, 0) # Take the derivative in x
abs_sobelx = np.absolute(sobelx) # Absolute x derivative to accentuate lines away from horizontal
scaled_sobel = np.uint8(255*abs_sobelx/np.max(abs_sobelx))
# Threshold x gradient
thresh_min = 20
thresh_max = 100
sxbinary = np.zeros_like(scaled_sobel)
sxbinary[(scaled_sobel >= thresh_min) & (scaled_sobel <= thresh_max)] = 1
# Threshold color channel
s_thresh_min = 170
s_thresh_max = 255
s_binary = np.zeros_like(s_channel)
s_binary[(s_channel >= s_thresh_min) & (s_channel <= s_thresh_max)] = 1
# Stack each channel to view their individual contributions in green and blue respectively
# This returns a stack of the two binary images, whose components you can see as different colors
color_binary = np.dstack(( np.zeros_like(sxbinary), sxbinary, s_binary))
# Combine the two binary thresholds
combined_binary = np.zeros_like(sxbinary)
combined_binary[(s_binary == 1) | (sxbinary == 1)] = 1
return combined_binary
Create a pipeline where all the above filters are applied¶
In [7]:
def pipeline_image(img):
# Gaussian Blur
kernel_size = 5
img = cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)
# Convert to HLS color space and separate the S channel
hls = cv2.cvtColor(img, cv2.COLOR_RGB2HLS)
s = hls[:,:,2]
# Grayscale image
gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
# Define sobel kernel size
ksize = 7
# Apply each of the thresholding functions
gradx = abs_sobel_thresh(gray, orient='x', sobel_kernel=ksize, thresh=(10, 255))
grady = abs_sobel_thresh(gray, orient='y', sobel_kernel=ksize, thresh=(60, 255))
mag_binary = mag_thresh(gray, sobel_kernel=ksize, mag_thresh=(40, 255))
dir_binary = dir_threshold(gray, sobel_kernel=ksize, thresh=(.65, 1.05))
# Combine all the thresholding information
combined = np.zeros_like(dir_binary)
combined[((gradx == 1) & (grady == 1)) | ((mag_binary == 1) & (dir_binary == 1))] = 1
# Threshold color channel
s_binary = np.zeros_like(combined)
s_binary[(s > 160) & (s < 255)] = 1
# Stack each channel to view their individual contributions in green and blue respectively
# This returns a stack of the two binary images, whose components you can see as different colors
color_binary = np.zeros_like(combined)
color_binary[(s_binary > 0) | (combined > 0)] = 1
# Defining vertices for marked area
imshape = img.shape
left_bottom = (100, imshape[0])
right_bottom = (imshape[1]-20, imshape[0])
apex1 = (610, 410)
apex2 = (680, 410)
inner_left_bottom = (310, imshape[0])
inner_right_bottom = (1150, imshape[0])
inner_apex1 = (700,480)
inner_apex2 = (650,480)
vertices = np.array([[left_bottom, apex1, apex2, \
right_bottom, inner_right_bottom, \
inner_apex1, inner_apex2, inner_left_bottom]], dtype=np.int32)
# Masked area
color_binary = region_of_interest(color_binary, vertices)
return color_binary
Display the images under different transformations¶
In [8]:
#img = cv2.imread('../CarND-Advanced-Lane-Lines/test_images/test2.jpg')
img=glob.glob('./test_images/test*.jpg')
for fname in img:
print(fname)
img = cv2.imread(fname)
undistorted=cal_undistort(img)
combined_color=color_transform(img)
combined_hls = pipeline_image(img)
f, (ax1, ax2, ax3, ax4) = plt.subplots(1, 4, figsize=(24, 9))
f.tight_layout()
distRGB = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
ax1.imshow(distRGB)
ax1.set_title('Original Image', fontsize=50)
undistRGB = cv2.cvtColor(undistorted,cv2.COLOR_BGR2RGB)
ax2.imshow(undistRGB)
ax2.set_title('Undistorted Image', fontsize=50)
undistRGB = cv2.cvtColor(undistorted,cv2.COLOR_BGR2RGB)
ax3.imshow(combined_color, cmap='gray')
ax3.set_title('Grad. Dir.', fontsize=50)
ax4.imshow(combined_hls, cmap='gray')
ax4.set_title('HLS', fontsize=50)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)
Apply a distortion correction to raw images¶
In [9]:
# Define image shape
image_shape = img.shape
# Define the region
area_of_interest = [[580,460],[710,460],[1150,720],[150,720]]
# Define a function that takes an image, number of x and y points,
# camera matrix and distortion coefficients
def unwarp(img, nx, ny, mtx, dist):
# Use the OpenCV undistort() function to remove distortion
undist = cv2.undistort(img, mtx, dist, None, mtx)
# Choose an offset from image corners to plot detected corners
offset1 = 200 # offset for dst points x value
offset2 = 0 # offset for dst points bottom y value
offset3 = 0 # offset for dst points top y value
# Grab the image shape
img_size = (gray.shape[1], gray.shape[0])
# For source points I'm grabbing the outer four detected corners
src = np.float32(area_of_interest)
# For destination points, I'm arbitrarily choosing some points to be
# a nice fit for displaying our warped result
dst = np.float32([[offset1, offset3],
[img_size[0]-offset1, offset3],
[img_size[0]-offset1, img_size[1]-offset2],
[offset1, img_size[1]-offset2]])
# Given src and dst points, calculate the perspective transform matrix
M = cv2.getPerspectiveTransform(src, dst)
Minv = cv2.getPerspectiveTransform(dst, src)
# Warp the image using OpenCV warpPerspective()
warped = cv2.warpPerspective(undist, M, img_size)
# Return the resulting image and matrix
return warped, M, Minv
Next step involves:¶
*Detecting lane pixels and fit to find the lane boundary.
*Determining the curvature of the lane and vehicle position with respect to center.
*Warping the detected lane boundaries back onto the original image.
In [10]:
# Define a class to receive the characteristics of each line detection
class Line():
def __init__(self):
# was the line detected in the last iteration?
self.detected = False
# x values of the last n fits of the line
self.recent_xfitted = []
#average x values of the fitted line over the last n iterations
self.bestx = None
#polynomial coefficients averaged over the last n iterations
self.best_fit = None
#polynomial coefficients for the most recent fit
self.current_fit = [np.array([False])]
#radius of curvature of the line in some units
self.radius_of_curvature = None
#distance in meters of vehicle center from the line
self.line_base_pos = None
#difference in fit coefficients between last and new fits
self.diffs = np.array([0,0,0], dtype='float')
#x values for detected line pixels
self.allx = None
#y values for detected line pixels
self.ally = None
#x values in windows
self.windows = np.ones((3,12))*-1
In [11]:
def find_curvature(yvals, fitx):
# Define y-value where we want radius of curvature
# I'll choose the maximum y-value, corresponding to the bottom of the image
y_eval = np.max(yvals)
# Define conversions in x and y from pixels space to meters
ym_per_pix = 30/720 # meters per pixel in y dimension
xm_per_pix = 3.7/700 # meteres per pixel in x dimension
fit_cr = np.polyfit(yvals*ym_per_pix, fitx*xm_per_pix, 2)
curverad = ((1 + (2*fit_cr[0]*y_eval + fit_cr[1])**2)**1.5) \
/np.absolute(2*fit_cr[0])
return curverad
def find_position(pts):
# Find the position of the car from the center
# It will show if the car is 'x' meters from the left or right
position = image_shape[1]/2
left = np.min(pts[(pts[:,1] < position) & (pts[:,0] > 700)][:,1])
right = np.max(pts[(pts[:,1] > position) & (pts[:,0] > 700)][:,1])
center = (left + right)/2
# Define conversions in x and y from pixels space to meters
xm_per_pix = 3.7/700 # meteres per pixel in x dimension
return (position - center)*xm_per_pix
In [12]:
def find_nearest(array,value):
# Function to find the nearest point from array
if len(array) > 0:
idx = (np.abs(array-value)).argmin()
return array[idx]
def find_peaks(image, y_window_top, y_window_bottom, x_left, x_right):
# Find the historgram from the image inside the window
histogram = np.sum(image[y_window_top:y_window_bottom,:], axis=0)
# Find the max from the histogram
if len(histogram[int(x_left):int(x_right)])>0:
return np.argmax(histogram[int(x_left):int(x_right)]) + x_left
else:
return (x_left + x_right) / 2
def sanity_check(lane, curverad, fitx, fit):
# Sanity check for the lane
if lane.detected: # If lane is detected
# If sanity check passes
if abs(curverad / lane.radius_of_curvature - 1) < .6:
lane.detected = True
lane.current_fit = fit
lane.allx = fitx
lane.bestx = np.mean(fitx)
lane.radius_of_curvature = curverad
lane.current_fit = fit
# If sanity check fails use the previous values
else:
lane.detected = False
fitx = lane.allx
else:
# If lane was not detected and no curvature is defined
if lane.radius_of_curvature:
if abs(curverad / lane.radius_of_curvature - 1) < 1:
lane.detected = True
lane.current_fit = fit
lane.allx = fitx
lane.bestx = np.mean(fitx)
lane.radius_of_curvature = curverad
lane.current_fit = fit
else:
lane.detected = False
fitx = lane.allx
# If curvature was defined
else:
lane.detected = True
lane.current_fit = fit
lane.allx = fitx
lane.bestx = np.mean(fitx)
lane.radius_of_curvature = curverad
return fitx
# Sanity check for the direction
def sanity_check_direction(right, right_pre, right_pre2):
# If the direction is ok then pass
if abs((right-right_pre) / (right_pre-right_pre2) - 1) < .2:
return right
# If not then compute the value from the previous values
else:
return right_pre + (right_pre - right_pre2)
# find_lanes function will detect left and right lanes from the warped image.
# 'n' windows will be used to identify peaks of histograms
def find_lanes(n, image, x_window, lanes, \
left_lane_x, left_lane_y, right_lane_x, right_lane_y, window_ind):
# 'n' windows will be used to identify peaks of histograms
# Set index1. This is used for placeholder.
index1 = np.zeros((n+1,2))
index1[0] = [300, 1100]
index1[1] = [300, 1100]
# Set the first left and right values
left, right = (300, 1100)
# Set the center
center = 700
# Set the previous center
center_pre = center
# Set the direction
direction = 0
for i in range(n-1):
# set the window range.
y_window_top = 720-720/n*(i+1)
y_window_bottom = 720-720/n*i
# If left and right lanes are detected from the previous image
if (left_lane.detected==False) and (right_lane.detected==False):
# Find the historgram from the image inside the window
left = find_peaks(image, y_window_top, y_window_bottom, index1[i+1,0]-200, index1[i+1,0]+200)
right = find_peaks(image, y_window_top, y_window_bottom, index1[i+1,1]-200, index1[i+1,1]+200)
# Set the direction
left = sanity_check_direction(left, index1[i+1,0], index1[i,0])
right = sanity_check_direction(right, index1[i+1,1], index1[i,1])
# Set the center
center_pre = center
center = (left + right)/2
direction = center - center_pre
# If both lanes were detected in the previous image
# Set them equal to the previous one
else:
left = left_lane.windows[window_ind, i]
right = right_lane.windows[window_ind, i]
# Make sure the distance between left and right laens are wide enough
if abs(left-right) > 600:
# Append coordinates to the left lane arrays
left_lane_array = lanes[(lanes[:,1]>=left-x_window) & (lanes[:,1]<left+x_window) &
(lanes[:,0]<=y_window_bottom) & (lanes[:,0]>=y_window_top)]
left_lane_x += left_lane_array[:,1].flatten().tolist()
left_lane_y += left_lane_array[:,0].flatten().tolist()
if not math.isnan(np.mean(left_lane_array[:,1])):
left_lane.windows[window_ind, i] = np.mean(left_lane_array[:,1])
index1[i+2,0] = np.mean(left_lane_array[:,1])
else:
index1[i+2,0] = index1[i+1,0] + direction
left_lane.windows[window_ind, i] = index1[i+2,0]
# Append coordinates to the right lane arrays
right_lane_array = lanes[(lanes[:,1]>=right-x_window) & (lanes[:,1]<right+x_window) &
(lanes[:,0]<y_window_bottom) & (lanes[:,0]>=y_window_top)]
right_lane_x += right_lane_array[:,1].flatten().tolist()
right_lane_y += right_lane_array[:,0].flatten().tolist()
if not math.isnan(np.mean(right_lane_array[:,1])):
right_lane.windows[window_ind, i] = np.mean(right_lane_array[:,1])
index1[i+2,1] = np.mean(right_lane_array[:,1])
else:
index1[i+2,1] = index1[i+1,1] + direction
right_lane.windows[window_ind, i] = index1[i+2,1]
return left_lane_x, left_lane_y, right_lane_x, right_lane_y
In [13]:
import math
# Function to find the fitting lines from the warped image
def fit_lanes(image):
# define y coordinate values for plotting
yvals = np.linspace(0, 100, num=101)*7.2 # to cover same y-range as image
# find the coordinates from the image
lanes = np.argwhere(image)
# Coordinates for left lane
left_lane_x = []
left_lane_y = []
# Coordinates for right lane
right_lane_x = []
right_lane_y = []
# Curving left or right - -1: left 1: right
curve = 0
# Set left and right as None
left = None
right = None
# Find lanes from three repeated procedures with different window values
left_lane_x, left_lane_y, right_lane_x, right_lane_y \
= find_lanes(4, image, 25, lanes, \
left_lane_x, left_lane_y, right_lane_x, right_lane_y, 0)
left_lane_x, left_lane_y, right_lane_x, right_lane_y \
= find_lanes(6, image, 50, lanes, \
left_lane_x, left_lane_y, right_lane_x, right_lane_y, 1)
left_lane_x, left_lane_y, right_lane_x, right_lane_y \
= find_lanes(8, image, 75, lanes, \
left_lane_x, left_lane_y, right_lane_x, right_lane_y, 2)
# Find the coefficients of polynomials
left_fit = np.polyfit(left_lane_y, left_lane_x, 2)
left_fitx = left_fit[0]*yvals**2 + left_fit[1]*yvals + left_fit[2]
right_fit = np.polyfit(right_lane_y, right_lane_x, 2)
right_fitx = right_fit[0]*yvals**2 + right_fit[1]*yvals + right_fit[2]
# Find curvatures
left_curverad = find_curvature(yvals, left_fitx)
right_curverad = find_curvature(yvals, right_fitx)
# Sanity check for the lanes
left_fitx = sanity_check(left_lane, left_curverad, left_fitx, left_fit)
right_fitx = sanity_check(right_lane, right_curverad, right_fitx, right_fit)
return yvals, left_fitx, right_fitx, left_lane_x, left_lane_y, right_lane_x, right_lane_y, left_curverad
In [14]:
# draw poly on an image
# def draw_poly(image, warped, yvals, left_fitx, right_fitx, Minv):
def draw_poly(image, warped, yvals, left_fitx, right_fitx,
left_lane_x, left_lane_y, right_lane_x, right_lane_y, Minv, curvature):
# Create an image to draw the lines on
warp_zero = np.zeros_like(warped).astype(np.uint8)
color_warp = np.dstack((warp_zero, warp_zero, warp_zero))
# Recast the x and y points into usable format for cv2.fillPoly()
pts_left = np.array([np.transpose(np.vstack([left_fitx, yvals]))])
pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, yvals])))])
pts = np.hstack((pts_left, pts_right))
# Draw the lane onto the warped blank image
cv2.fillPoly(color_warp, np.int_([pts]), (0, 255, 0))
# Warp the blank back to original image space using inverse perspective matrix (Minv)
newwarp = cv2.warpPerspective(color_warp, Minv, (image.shape[1], image.shape[0]))
# Combine the result with the original image
result = cv2.addWeighted(image, 1, newwarp, 0.3, 0)
# Put text on an image
font = cv2.FONT_HERSHEY_SIMPLEX
text = "Radius of Curvature: {} m".format(int(curvature))
cv2.putText(result,text,(400,100), font, 1,(255,255,255),2)
# Find the position of the car
pts = np.argwhere(newwarp[:,:,1])
position = find_position(pts)
if position < 0:
text = "Vehicle is {:.2f} m left of center".format(-position)
else:
text = "Vehicle is {:.2f} m right of center".format(position)
cv2.putText(result,text,(400,150), font, 1,(255,255,255),2)
return result
In [15]:
def process_image(image):
# Apply pipeline to the image to create black and white image
img = pipeline_image(image)
# Warp the image to make lanes parallel to each other
top_down, perspective_M, perspective_Minv = unwarp(img, 6, 9, mtx, dist)
# Find the lines fitting to left and right lanes
a, b, c, lx, ly, rx, ry, curvature = fit_lanes(top_down)
# Return the original image with colored region
return draw_poly(image, top_down, a, b, c, lx, ly, rx, ry, perspective_Minv, curvature)
Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position¶
In [16]:
# These are to be used to plot lines on images
x_values = [area_of_interest[0][0],area_of_interest[1][0],area_of_interest[2][0],area_of_interest[3][0],area_of_interest[0][0]]
y_values = [area_of_interest[0][1],area_of_interest[1][1],area_of_interest[2][1],area_of_interest[3][1],area_of_interest[0][1]]
img=glob.glob('./test_images/test*.jpg')
for fname in img:
print(fname)
# Set up lines for left and right
left_lane = Line()
right_lane = Line()
# load the image
img_raw = cv2.imread(fname)
# Apply pipeline to the image to create black and white image
img = pipeline_image(img_raw)
img_size=img.shape[0:2]
img=cal_undistort(img)
# Unwrap the image
top_down, perspective_M, perspective_Minv = unwarp(img, 6, 9, mtx, dist)
# Set up for the subplots
f, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(24, 9))
f.tight_layout()
# Showing the image from pipeline with marked areas
ax1.set_title('Gray Image', fontsize=20)
ax1.plot(x_values,y_values,'r-',lw=2)
ax1.imshow(img, cmap='gray')
# Find lanes from the warped image
a, b, c, _, _, _, _, _ = fit_lanes(top_down)
ax2.plot(b, a, color='red', linewidth=5)
ax2.plot(c, a, color='red', linewidth=5)
ax2.imshow(top_down, cmap='gray')
ax2.set_title('Undistorted and Warped Image', fontsize=20)
# Set up lines for left and right
left_lane = Line()
right_lane = Line()
# Use the information from fit_lane function to color the lanes
image_color = process_image(img_raw)
ax3.imshow(image_color)
ax3.set_title('Image with a color', fontsize=20)
plt.subplots_adjust(left=0., right=1, top=0.9, bottom=0.)
In [17]:
### Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
# Set up lines for left and right
left_lane = Line()
right_lane = Line()
white_output = 'project_video_out.mp4'
clip1 = VideoFileClip("project_video.mp4")
white_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)
Briefly discuss any problems / issues you faced in your implementation of this project. Where will your pipeline likely fail?¶
One of the main concerns related to the project is that is not fast - creating the video took almost 10 min and I believe that the pipeline is a botleneck in this case. Some of the improvements that can be done is regarding the finding the lanes and having faster way on detecting them. I am not sure what the histogram can be replaced with, or how some of the sanity check can help in the process. The accuracy in finding the lane is not bad for the challenge video, however for harder_challenge_video improvements are needed.
In [ ]: